Enabling multi-modal search for inspirational design stimuli using deep learning

Abstract Inspirational stimuli are known to be effective in supporting ideation during early-stage design. However, prior work has predominantly constrained designers using text-only queries when searching for stimuli, which is not consistent with real-world design behavior where fluidity across modalities (e.g., visual, semantic, etc.) standard practice. In the current work, we introduce a multi-modal search platform that retrieves inspirational form of 3D-model parts text, appearance, and function-based inputs. Computational methods leveraging deep-learning approach presented designing this platform, relies on deep-neural networks trained large dataset parts. This further presents results cognitive study ( n = 21) aforementioned was used find inspire solutions challenge. Participants engaged three different modalities: by keywords, 3D parts, user-assembled their workspace. When selected or workspace, participants had additional control over similarity appearance function relative input. The demonstrate modality impacts behavior, such as frequency, how retrieved with, broadly space covered. Specific link interactions interface strategies may have task. Findings suggest desired can achieved both direct inputs keyword) well more randomly discovered examples, specific goal defined. Both processes found important enable platforms retrieval.